Experimental Design 


Experiments are used to study causal relationships (rather then 
“just” correlations as you can infer from observational studies) 


So what are some key characteristics of a good experimental 
design? 


Experimental Design 


e Context: 
= Experiment is embedded in theory 
= Provides results that are easy to interpret 
= Permits conclusions that have wide validity (large sphere of inference) 


e Design 
= Provides an unbiased estimate of effects 
= Provides information for every outcome (good effect size) 


e Analysis 
€" Is clearly linked to a statistical treatment 


Context 


What is your research question? 


How does it fit with theory? 


Consider your variables and how they are related 


What are you going to measure? 


Write a specific, testable hypothesis 


= Brief, clear, precise, and definitive 
= States an expected relationship between two or more variables 
= Testable 


Context - Why is it important that it fits within a larger theory? 


There is no actual | There is an actual 
difference difference 


Accept null Type Il error 
Statistical errors hypothesis of no | Correct conclusion | Accept incorrect Ho 
difference False negative 


Type | error 


Reject correct Ho orrect conclusion 
False positive 


Reject null 
hypothesis 


Context - Why is it important that it fits within a larger theory? 


What is the chance of a false positive with a=0.05 (or 5%)? 


The expectation is that 1/20 experiments where there is NO underlying difference 
between the treatment/populations will produce a false positive. 


However, if we have a good theoretical expectation, most of our treatments are 
expected to produce different results. So overall the percentage of false positives is 
much lower than 1/20! 


But if you randomly test variables, you will get 1/20 false positives. 


Experimental Design 


Key elements: 


Control/comparison group 


- Acontrol is designed to isolate the variable of interest. It should assure that only 
the tested variable changes between the experimental and comparison group 


- Negative/positive control 
Replication 
Randomisation 


Experimenter bias 


Experimental Design 


Key elements: 


Control/comparison group 

Replication = is the repetition of an experimental condition 
- Reduce and estimate variation 

- Experimental unit / independence 

Randomisation 


Experimenter bias 


Experimental Design 


Key elements: 

e Control/comparison group 

e Replication 

e Randomisation 
- eliminates bias 
- balances the groups with respect to known and unknown confounding variables 
- blocking 


e Experimenter bias 


Experimental Design — blocking 
assign 10 animals to two treatments 
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Experimental Design — latin square designs 
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A latin square allows the control/balance of external sources of variation - particularly 
temporal effects or sequence effects in repeated-measures designs (e.g. animals that are 
learning) 


Latin squares assure that each animal (e.g. columns) see each treatment (e.g. rows) once 
and each treatment is seen once first, once second.....by an animal 


Subjects and treatments are randomly allocated to columns and rows 


Experimental Design 


Key elements: 

e Control/comparison group 
e Replication 

e Randomisation 


e Experimenter bias 
- Much of the experimenter bias has already been eliminated by the above 
randomisation 
- Blind and double blind experiments are state of the art treatments to reduce the 
effects even further 


Analysis 


The analysis should reflect the experimental design and vice versa. 
The most important issue to consider is the independence of replicas 


e Blocking 
e Repeated measures 
e Temporal sequences 


All affect the independence of replicas and have to be taken into account to avoid 
pseudo replication (e.g. repeated measures on the same animal) 


Pseudoreplication 


Which of these designs is ok and which amounts to pseudoreplication when analysed with a simple t-test 
comparing A with B? All are sampled over the course of a day (consecutive rows) 
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